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EXECUTIVE  SUMMARY 


Until  recently,  little  attention  has  been  paid  to  the  way  written  procedures  which  are  used  to 
develop  and  revise  aircraft  maintenance  technical  data  affect  the  users  of  that  data.  Studies  of 
maintenance  problems  have  tended  to  focus  on  the  actions  of  the  mechanic,  job  culture,  and  work 
procedures.  More  recently,  attempts  have  been  made  to  document  the  source  of  maintenance 
errors  and  improve  maintenance  procedures.  One  of  the  identified  contributing  causes  of  errors 
is  the  documentation  used  to  guide  maintenance  tasks.  As  a  result,  efforts  have  been  made  to 
establish  guidelines  for  the  design  of  maintenance  job  aids.  A  question  that  remains  is  how  the 
procedures  used  by  manufacturers  to  develop  maintenance  data  may  contribute  to  user  error. 

This  report  contains  the  results  from  Phase  2  of  a  3-phase  research  effort.  This  phase 

(1)  examines  the  procedures  used  by  industry  to  develop  aircraft  maintenance  manuals, 

(2)  documents  the  problems  encountered  by  the  users  of  these  documents,  and  (3)  identifies  ways 
in  which  human  factors  principles  can  be  used  to  improve  the  development  of  these  documents. 
This  phase  includes  a  survey  and  interviews  of  technicians  responsible  for  maintenance  of  a  wide 
variety  of  Federal  Aviation  Regulations  Part  25  aircraft.  Technicians  were  queried  about 
technical  manual  usage  rates,  manual  errors,  general  manual  quality,  potential  safety  impact  of 
manual  problems,  and  suggestions  for  manual  improvement.  In  summary,  the  results  show  that 
the  accuracy  and  quality  of  technical  manuals  are  rated  as  being  good  but  have  poor  usability.  In 
light  of  the  results  of  Phase  1,  these  results  support  the  need  for  a  greater  degree  of  user 
involvement  during  the  document  development  process. 
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1.  INTRODUCTION. 


The  Federal  Aviation  Administration  (FAA)  has  committed  themselves  to  the  goal  of  reducing 
fatal  aircraft  accidents  by  80%  (based  on  1996  baseline  rate)  by  the  year  2007  [1].  An  important 
part  of  accident  reduction  is  to  reduce  the  number  of  errors  generated  during  the  maintenance  of 
aircraft.  In  an  analysis  of  aircraft  maintenance  error  causation,  Johnson  and  Watson  [2] 
identified  information  as  being  the  highest  ranked  contributing  cause,  which  was  implicated  in 
approximately  38%  of  all  maintenance  errors.  An  analysis  of  National  Aeronautics  and  Space 
Administration  (NASA)  Aviation  Safety  Reporting  System  data,  regarding  maintenance 
incidents,  found  document  procedures  to  be  related  to  60%  of  incident  reports  from  1986  to  1992 
and  45%  of  incidents  from  1996  to  1997  [3].  Further  analysis  of  the  errors  attributed  to 
information  revealed  that  incorrect  data  was  a  factor  in  only  a  small  number  of  these  cases,  and 
many  of  those  cases  were  user-initiated  problems.  The  most  common  problem  appeared  in  cases 
where  the  information  was  not  being  referred  to,  misunderstood,  or  disregarded  in  favor  of  an 
alternate  method  of  performing  a  maintenance  procedure.  Given  the  number  of  cases  in  which 
technicians  fail  to  properly  use  maintenance  information;  one  might  conclude  that  the  problem 
should  be  addressed  through  training  or  disciplinary  action  toward  the  maintenance  workforce. 
However,  rather  than  indicating  a  systemic  discipline  problem  with  maintenance  technicians,  or 
laissez-faire  attitude  toward  technical  documents,  it  could  reflect  a  problem  with  the  usability  of 
technical  documents. 

Maintenance  manuals  can  contribute  to  maintenance  error  if  they  contain  misleading  information, 
insufficient  information,  or  unclear  procedures.  Not  only  must  the  information  be  technically 
sound,  it  must  also  be  presented  in  an  effective  manner.  A  term  common  to  the  computer 
industry,  and  most  applicable  in  this  case,  is  “usability.”  Usability  can  be  defined  in  a  broad 
sense  as  “the  ease  with  which  a  user  can  learn  to  operate,  prepare  inputs  for,  and  interpret  outputs 
of  a  system  or  component”  [4].  When  applied  to  aircraft  manuals,  usability  includes  the  user’s 
experience  with  manuals;  how  easy  they  are  to  use,  how  well  they  match  the  technician’s 
representation  of  a  task,  how  easy  they  are  to  read  and  interpret,  and  the  usefulness  of  the 
information  they  contain.  The  amount  of  experience  the  user  has  had  with  manuals  influences 
how  they  use  them  and  the  degree  to  which  they  refer  to  them. 

1.1  BACKGROUND. 

One  of  the  problems  identified  in  Phase  1  of  this  effort  [5]  was  the  lack  of  a  systematic  method 
for  the  evaluation  of  manual  quality  by  manufacturers.  Evaluations  may  consist  of  peer  review, 
grammar  and  spell  checking,  and  critical  reviews  of  layouts  to  assure  adherence  to  standards.  All 
of  these  procedures  are  necessary  components  of  the  error-checking  process.  A  fundamental  step 
in  the  development  of  a  quality  technical  document  is  to  determine  that  the  intended  message  is 
presented  clearly  and  without  error.  The  results  of  the  user  surveys  indicate  that,  for  the  most 
part,  manufacturers  have  done  a  good  job  in  this  area.  Not  only  does  the  summary  of  responses 
reflect  a  favorable  evaluation  of  the  quality  and  consistency  of  documents  by  technicians,  but  the 
evaluations  are  remarkably  similar  when  analyzed  by  individual  manufacturers.  While  the  actual 
error  rate  is  difficult  to  determine,  it  is  not  of  sufficient  size  to  create  a  negative  perception  on  the 
part  of  technicians. 
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Compliance  with  standards  for  design  and  development  insure  that  the  technical  documents  have 
a  consistent  “look  and  feel.”  Design  standards  are  defined  by  company  policies  and  procedures, 
or  may  be  part  of  industrywide  efforts  such  as  the  Air  Transport  Association  (ATA)  formatting 
specification.  Systematic  procedures  for  verifying  the  accuracy  of  technical  information  reduce 
the  number  of  errors  (typographical  and  factual).  Adherence  to  standards  can  be  accomplished 
through  oversight  within  the  organization,  or  it  may  be  more  formal  as  in  the  ISO  9000/9001 
certification.  However,  adherence  to  company  procedures  and  design  standards  alone  does  not 
guarantee  that  the  product  will  be  easy  to  use. 

1.2  PROJECT  OBJECTIVES- 

Phase  1  of  this  research  effort  surveyed  five  manufacturers  concerning  the  procedures  they  use  to 
develop  maintenance  documentation.  Several  human  factors  issues  were  identified  in  the 
development  process  employed  by  these  manufacturers.  They  included  the  reactive  rather  than 
proactive  use  of  user  evaluations,  the  limited  use  of  user  input  and  procedure  validation,  no 
systematic  attempts  to  track  errors,  and  the  lack  of  standards  for  measuring  document  quality. 
Given  the  issues  identified  in  Phase  1,  the  objective  of  Phase  2  was  to  gather  information  about 
errors  in  technical  documents,  manual  usage  rates,  and  user  perceptions  of  manual  quality. 
Respondents  were  also  asked  to  identify  the  types  of  problems  encountered  with  technical 
documents,  the  impact  of  those  problems,  and  suggestions  for  improving  manuals.  Information 
was  gathered  on  the  differences  between  manuals  developed  by  different  companies. 
Considering  the  variability  of  methods  used  by  manufacturers  to  develop  technical  documents,  it 
may  be  possible  to  identify  techniques  and  procedures  that  result  in  more  effective 
documentation.  The  goal  of  Phase  3  will  be  to  outline  recommendations  and  procedures  for 
improving  document  usability. 

2.  METHODS. 

2.1  SURVEY. 

2.1.1  Recruiting  Participants. 

Phase  1  of  this  effort  focused  on  the  procedures  used  by  manufacturers  to  develop  and  revise  the 
technical  documentation  to  be  used  in  the  maintenance  of  their  aircraft.  In  order  to  determine  the 
potential  effects  of  these  procedures  on  document  quality  and  usability,  a  survey  was  developed 
to  measure  technician  perceptions  of  maintenance  documents.  In  addition  to  general  perceptions 
of  documentation  quality  and  usability,  respondents  were  asked  to  compare  the  documentation 
produced  by  different  manufacturers  so  that  differences  in  user  evaluation  might  be  traced 
directly  to  the  procedures  used  by  a  particular  manufacturer. 

In  order  to  gather  the  most  representative  sample  possible,  participation  was  solicited  from 
operators  and  facilities  responsible  for  the  maintenance  of  a  variety  of  Federal  Aviation 
Regulations  (FAR)  Part  25  aircraft.  Agreement  to  participate  was  first  obtained  from  the 
corporate  offices  of  the  aircraft  operator.  Eleven  facilities  providing  maintenance  services  for 
regional  and  privately  owned  FAR  Part  25  aircraft  were  contacted  for  the  study  and  agreed  to 
participate.  Seven  facilities  responsible  for  providing  maintenance  on  large  FAR  Part  25  aircraft 
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were  also  contacted,  and  two  agreed  to  participate.  Two  other  companies  deferred  participation 
to  a  later  date,  and  that  data  will  be  included  in  a  subsequent  report.  Following  corporate 
approval,  the  associated  local  labor  union  representatives  were  contacted,  provided  with  a 
description  of  the  intent  and  the  purpose  of  the  project,  and  asked  to  participate  in  the  project. 
Participants  were  informed  that  all  information  was  confidential  and  that  they  would  not  be 
identified  in  any  of  the  reports.  The  majority  of  completed  surveys  for  large  FAR  Part  25  aircraft 
were  obtained  from  the  maintenance  facilities  operated  by  one  major  airline,  and  a  company 
providing  maintenance,  repair,  and  overhaul  services  to  airlines. 

2.1.2  Survey  Description. 

The  survey  (see  appendix  A)  solicited  information  from  the  respondents  about  a  number  of  areas, 
including  the  aircraft  they  currently  work  on  and  their  specialty  area.  Participants  were  then 
asked  to  identify  the  two  aircraft  with  which  they  were  most  familiar,  and  to  compare  the 
frequency  of  errors,  perceptions  of  quality,  satisfaction,  and  usability,  of  those  manuals.  Next, 
the  respondents  were  asked  to  report  how  frequently  they  had  engaged  in  safety  related 
maintenance  behaviors,  and  to  identify  the  consequences  of  errors  or  confusing  information  they 
have  encountered  in  the  manuals.  Finally,  information  was  solicited  about  document  format 
preferences  and  suggestions  for  improving  the  manuals.  Biographical  information,  including 
education  level  and  employment  experience,  was  collected  for  the  purpose  of  identifying  any 
trends  in  user  responses  that  may  have  been  due  to  individual  differences.  Responses  to 
assessments  and  ratings  of  the  manual  were  reported  using  a  5-point  Likert  scale.  Other  data  was 
recorded  as  discrete  or  narrative  responses  when  appropriate. 

2.1.3  Survey  Distribution. 

A  designated  person  at  each  company  was  responsible  for  distributing  and  collecting  completed 
surveys  from  employees.  The  survey  was  available  as  a  four-page  paper  document  and  as  a  web- 
based  form  (www.niar.twsu.edu/faaq)  to  increase  the  scope  of  distribution.  Both  paper  and  web- 
based  surveys  were  accompanied  by  a  cover  letter  describing  the  purpose  of  the  project  and 
emphasized  the  confidential  nature  of  the  responses. 

2.2  INTERVIEWS. 

In  addition  to  the  questionnaire,  site  visits  were  made  to  one  regional  operator  and  two  major 
airline  facilities.  The  employees  who  use  the  manuals  at  these  facilities  were  interviewed  to 
verify  the  reliability  and  validity  of  survey  responses,  as  well  as  providing  additional  detail  and 
clarification  of  the  survey  data.  Interviews  were  conducted  with  personnel  involved  in  all  areas 
of  aircraft  maintenance,  including  technicians,  supervisors,  engineers,  parts  supply,  and  task  card 
writers.  Interview  participants  were  asked  to  identify  areas  of  strength  and  weakness  in  the 
technical  documentation  provided  by  manufacturers,  to  compare  and  contrast  different 
manufacturers,  to  assess  the  potential  impact  of  document  quality,  and  to  make  suggestions  for 
any  improvements  that  could  be  made  to  manufacturers’  documents.  Whenever  possible, 
interviews  were  conducted  in  groups  of  twos  or  threes  to  facilitate  discussion  and  to  limit  peer 
influence  that  might  be  present  in  a  larger  group.  As  with  the  surveys,  agreement  to  participate 
was  obtained  from  the  corporate  offices  of  the  aircraft  operator  and  the  local  labor  union 
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representatives.  All  interview  participants  were  informed  that  the  information  they  provided 
would  be  confidential.  They  were  also  assured  that  names  of  individuals  and  their  respective 
companies  would  not  be  identified  in  any  of  the  reports. 

3.  RESULTS. 

3.1  SURVEY  RESPONSE 

Completed  surveys  included  feedback  from  technicians  at  both  line-  and  heavy-based 
maintenance  facilities.  Figure  1  shows  the  manufacturers  of  the  aircraft  that  technicians 
maintain,  and  their  relative  frequency.  The  most  frequently  cited  manufacturers  were  Boeing 

^,rdV^a\ SSSnf-  ^  AlfBuS'  The  majority  of  resP°nses  came  from  major  airline 
facilities.  To  date,  377  individual  survey  responses  and  745  unique  aircraft  evaluations  have 

been  received.  Of  these  responses,  the  296  individual  survey  responses  and  579  unique  aircraft 
evaluations  came  from  major  aircraft  operators.  The  remaining  responses  came  from 
maintenance  facilities  responsible  for  regional  and/or  privately  owned  FAR  Part  25  aircraft. 
Figure  2  shows  the  relative  percentage  of  responses  for  large  and  smaller  FAR  Part  25  aircraft 
respectively.  The  relative  proportion  of  different  aircraft  types  reported  in  the  survey  appears  to 
be  representative  of  the  number  in  active  service. 


FIGURE  1.  AIRCRAFT  MANUFACTURERS  REPRESENTED 
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Large  Aircraft  Models  Smaller  Aircraft  Models 


FIGURE  2.  AIRCRAFT  MODELS  REPRESENTED  (Large  and  smaller) 


3.2  DEMOGRAPHICS. 

A  summary  of  the  demographic  characteristics  of  the  survey  respondents  is  shown  in  table  1. 
The  results  reveal  that  the  sample  is  almost  exclusively  male,  is  highly  experienced,  and  consists 
of  almost  equal  numbers  of  line  and  base  maintenance  personnel,  which  the  vast  majority  are 
FAA  certified. 


TABLE  1.  RESPONDENT  DEMOGRAPHICS 


Mean  Age 

The  average  age  of  respondents  was  45  years,  with  the  distribution 
ranging  from  22  to  69  years.  Age  was  normally  distributed,  with  the 
bulk  of  responses  coming  from  technicians  39  to  46  years  of  age. 

Sex 

With  the  exception  of  a  single  technician,  all  respondents  were  male. 

Education 

The  majority  of  respondents  (54%)  indicated  having  some  college 
education.  12.5%  report  having  a  Bachelor’s  degree  or  higher. 

Certification 

92%  of  all  respondents  and  94%  of  respondents  working  for  major 
airlines  were  FAA  certificated  maintenance  technicians. 

Type  of 
Maintenance 

51%  of  responses  were  from  technicians  responsible  for  Line 
Maintenance,  while  the  remaining  49%  were  responsible  for  Base 
Maintenance,  or  a  combination  of  Line  and  Base  Maintenance. 

Experience 

The  average  work  experience  reported  was  14  years  as  a  technician,  and 

13  years  with  their  current  company. 

Military 

Training 

58%  of  respondents  report  some  military  experience. 

3.3  SURVEY  QUESTIONS. 


The  first  group  of  survey  questions  addressed  the  usage  rates  and  the  perception  of  quality  of 
manuals.  Respondents  were  asked  to  identify  the  two  aircraft  types  they  are  most  familiar  with 
and  compare  them  on  each  set  of  ratings. 

3.3.1  Manual  Usage. 

Responses  indicated  that  the  Aircraft  Maintenance  Manual  (AMM),  Illustrated  Parts  Catalog,  and 
task  cards  were  the  most  frequently  used  technical  documents.  It  should  be  acknowledged  that 
because  the  survey  sample  included  a  large  cross-section  of  maintenance  operations,  the  technical 
documentation  used  by  any  particular  technician  will  differ  depending  on  whether  they  are  doing 
line  or  base  maintenance,  and  the  degree  to  which  they  specialize  in  specific  maintenance  tasks. 
For  example,  scheduled  base  maintenance  will  rely  more  heavily  on  the  use  of  task  cards,  while 
the  unpredictable  nature  of  line  maintenance  requires  the  maintenance  manual  and  parts  catalog 
to  be  used  more  frequently.  It  should  be  emphasized  that  the  focus  of  this  survey  and  evaluation 
was  the  AMM  and  not  job  task  cards.  Interested  readers  should  consult  the  work  of  Drury  and 
colleagues  [6]  which  focuses  on  the  unique  design  and  usability  issues  associated  with  task  cards. 

3.3.2  Error  in  Manuals. 

After  establishing  manual  usage  rates,  technicians  were  asked  to  report  how  often  they  found 
errors  in  text  procedures,  illustrations,  and  diagrams..  One  of  the  larger  goals  of  this  project  was 
to  determine  the  degree  of  error  present  in  technical  manuals.  Respondents  were  asked  to  rate, 
on  a  5-point  Likert  scale,  from  “never”  to  “very  often”  how  often  they  have  encountered  errors  in 
manual  text,  diagrams,  and  procedures.  While  subjective,  the  results  (see  figure  3)  show  that 
most  respondents  reported  rarely  or  never  finding  errors  in  manual  text  (54%),  illustrations 
(63%),  and  diagrams  (66%). 

Similar  results  are  observed  when  responses  are  separated  by  aircraft  size  (e.g.,  aircraft  operated 
by  major  airlines  versus  smaller  FAR  Part  25  aircraft).  A  statistical  comparison  of  large  and 
smaller  aircraft  reveals  significant  differences  only  in  user  ratings  of  the  frequency  of  text  errors, 
with  error  being  reported  more  frequently  in  smaller  FAR  Part  25  aircraft.  This  comparison  can 
be  seen  in  figure  4.  A  complete  copy  of  the  statistical  analysis  and  results  are  available  in 
appendix  B.  ” 


6 


Percent 


Found  Error  in  Text 


60 


50 


40 


30 


20 


10 


0 

Never  Rarely  Occasionally  Often  Very  Often 


Found  Error  in  Illustrations 

60 


50 


40 


S  30 
d> 

Q_ 

20 


10 


0 

Never  Rarely  Occasionally  Often  Very  Often 


Found  Error  in  Diagrams 


Never  Fiarely  Occasionally  Often  Very  Often 


FIGURE  3.  ERROR  REPORTED  FOR  ALL  AIRCRAFT 
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FIGURE  4.  ERROR  REPORTED  FOR  LARGE  VERSUS  SMALLER  AIRCRAFT 


3.3.3  General  Manual  Quality. 


The  next  series  of  questions  on  the  survey  queried  the  user’s  perception  of  general  manual 
quality.  Respondents  were  asked  to  rate  from  “Very  poor”  to  “very  good”  on  the  usefulness  of 
manuals,  the  quality  of  manuals  and  diagrams,  and  the  clarity  of  text  descriptions.  Figure  5 
shows  that  the  technicians  rate  maintenance  manuals  as  being  very  useful  to  the  job  they  perform 
(74%  “good”  or  “very  good”)  and  perceive  the  manuals  as  being  of  generally  good  quality  (56%) 
with  good  quality  diagrams  (54%)  and  text  clarity  (47%). 
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FIGURE  5.  PERCEPTION  OF  GENERAL  MANUAL  QUALITY 

In  addition  to  distinctions  between  large  and  smaller  aircraft  manufacturers,  it  is  important  to 
investigate  potential  differences  between  manufacturers  of  similar  aircraft.  If  one  manufacturer 
is  consistently  rated  higher  than  their  competitors,  this  may  be  indicative  of  superior  manual 
development  techniques.  Figure  6  represents  a  sample  of  data  from  the  four  large  aircraft 
manufacturers  referred  to  most  often  in  this  survey.  Closer  scrutiny  of  these  figures  reveals  that 
the  quality  of  the  manuals  is  roughly  comparable  for  three  of  these  manufacturers,  while  the 
fourth  is  noticeably  worse.  From  these  results,  it  appears  that  even  though  each  manufacturer 
employs  different  manual  development  procedures,  most  result  in  manuals  that  are  perceived  to 
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be  of  similar  quality.  The  poorer  ratings  of  manufacturer  D  may  be  accounted  for  by  the  fact  that 
this  particular  manual  was  translated  into  English  and  is  reported  to  be  a  paper-based  version  of 
what  was  originally  intended  to  be  an  electronic  manual  (i.e.,  CD-ROM). 
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FIGURE  6.  EXAMPLE  OF  DIFFERENCES  IN  PERCEPTIONS  OF  TEXT 
CLARITY  BY  MANUFACTURER 


3.3.4  Usability. 

The  next  set  of  survey  questions  were  designed  to  investigate  user  perceptions  of  manual 
usability.  Usability  issues  surveyed  included  the  ease  of  manual  use,  the  consistency  and  clarity 
of  the  manual,  and  the  depth  of  information.  The  respondent  answers  shown  in  figures  7(a)  and 
7(b)  are  normally  distributed  with  the  mean  centered  near  neutral  or  shifted  slightly  toward  the 
positive  end  of  the  scale.  In  general,  technicians  consider  maintenance  manuals  easy  to  learn, 
easy  to  use,  clear,  and  consistent. 
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FIGURE  7.  MEASURES  OF  MANUAL  USABILITY 


The  final  two  usability  questions  assess  how  well  the  described  procedures  match  the  way 
technicians  actually  do  their  job.  When  asked  to  respond  to  the  statement,  “the  manual  describes 
the  best  way  to  do  a  procedure,”  47%  of  technicians  say  they  disagree  or  strongly  disagree.  The 
statement,  “the  manual  writer  understands  the  way  I  do  maintenance,”  resulted  in  54%  of 
technicians  responding  that  they  disagree  or  strongly  disagree. 
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FIGURE  7.  MEASURES  OF  MANUAL  USABILITY  (Continued) 


3.3.5  Impact  of  Manual  Usability. 

Figures  8(a)  and  8(b)  show  the  results  of  questions  designed  to  assess  the  potential  impact  of 
document  usability.  As  can  be  seen  in  figure  8(a),  62%  of  the  respondents  reported  completing  a 
procedure  in  a  way  they  considered  “better”  than  what  was  described  in  the  manual.  As  a  result 
of  unclear  or  misleading  procedures,  18%  of  the  respondents  reported  parts  being  damaged,  20% 
reported  assembling  a  component  incorrectly,  and  25%  reported  having  adjusted  or  rigged  a 
system  incorrectly. 
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FIGURE  8.  REPORTED  IMPACT  OF  USABILITY  PROBLEMS 

Seventy-eight  percent  of  respondents  reported  that  they  consulted  another  technician  when 
confronted  with  a  confusing  procedure  and  64%  reported  finding  their  own  way  of  performing  a 
procedure.  Only  5%  reported  that  these  difficulties  resulted  in  an  aircraft  being  released  in 
unairworthy  condition,  but  30%  reported  seeing  a  technician  being  blamed  for  a  mishap 
stemming  from  difficulties  interpreting  the  manual. 
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FIGURE  8.  REPORTED  IMPACT  OF  USABILITY  PROBLEMS  (Continued) 


In  order  to  determine  the  effectiveness  of  the  procedures  intended  to  provide  feedback  to 
manufacturers  about  the  problems  users  encounter  within  the  manual,  respondents  were 
questioned  about  how  likely  a  technician  is  to  report  an  identified  error.  The  results  presented  in 
figure  9  indicate  that  45%  of  respondents  say  they  will  often  or  very  often  report  errors  when  they 
find  them.  Fifty-three  percent  reported  that  they  report  the  errors  they  find  only  occasionally, 
rarely,  or  never. 
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When  Manual  Problems  are  Found, 
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FIGURE  9.  REPORTING  IDENTIFIED  PROBLEMS 
3.4  TECHNICIAN  COMMENTS. 

The  survey  contained  a  section  wherein  the  participants  were  asked  open-ended  questions  about 
their  preferred  manual  delivery  mode  (CD,  paper  manuals,  and  microfiche)  and 
recommendations  for  improving  the  manuals.  These  results  are  discussed  below. 

3.4.1  Format  Preference. 

Although  this  research  effort  does  not  address  the  unique  differences  between  manual  formats,  it 
is  important  to  acknowledge  the  usability  issues  associated  with  different  delivery  formats  (e.g., 
CD,  microfiche,  and  paper  manuals).  For  example,  the  problems  of  determining  effectivity  of 
parts  and  systems  modifications  can  be  more  easily  addressed  in  electronic  format  manuals  by 
linking  the  aircraft  registration  number  to  a  database  of  parts  and  systems  modifications. 
However,  the  use  of  electronic  manuals  raises  usability  concerns  of  interface  and  display  design, 
information  presentation,  computer  accessibility  and  portability,  and  system  reliability.  Clearly, 
it  is  not  sufficient  to  simply  present  the  same  information  in  an  electronic  form. 

In  the  short  term,  there  is  a  potential  concern  about  the  willingness  of  technicians  to  embrace  the 
use  of  computerized  manuals.  To  explore  this  concern,  the  survey  included  questions  about  what 
manual  format  was  preferred  and  why.  The  overwhelming  majority  of  respondents  (68%) 
indicated  a  preference  for  CD-ROM  based  or  other  electronic  manual  formats.  Most  technicians 
cite  the  speed  of  information  search  in  their  preference.  The  second  most  common  preference 
(18%)  was  paper,  with  most  respondents  citing  its  portability  as  being  advantageous.  The 
willingness  of  technicians  to  leam  to  use  computer  manuals  does  not  appear  to  be  a  concern  and 
was  not  mentioned  by  any  of  the  respondents. 
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3.4.2  Suggestions  for  Improvement. 


Finally,  technicians  were  asked  to  make  recommendations  for  improving  maintenance 
documentation.  In  some  cases,  responses  reflected  local  problems  within  their  own  organization. 
These  responses  included  complaints  of  limited  access  to  computers  or  poor  quality  viewing  and 
printing  equipment  that  made  the  reproduced  manual  figures  or  text  difficult  to  read.  As  such, 
these  complaints  are  a  reflection  of  the  degree  of  investment  in  technology  to  support 
maintenance  activities  rather  than  of  manual  usability. 

The  most  common  suggestion  was  to  expand  the  use  of  electronic  manuals  and  the  features  they 
contain.  Respondents  commented  that  they  would  like  to  see  more  extensive  use  of  electronic 
documents  and  a  better  utilization  of  the  capabilities  of  computers  through  video  and  audio 
information,  improved  search  capabilities,  and  more  extensive  linking  of  diagrams,  illustrations, 
and  parts  catalogs. 

•  Make  all  reference  material  available  on  CD-ROM  with  search  software  implemented  in 
all  programs” 

•  “Ability  to  cross-reference  parts  with  procedures” 

•  “Digitize  all  schematics  and  wiring  diagrams” 

•  “All  should  be  on  CD-ROM  with  better  graphics  ability” 

The  second  most  cited  suggestion  supports  the  fact  that  technicians  do  not  have  a  problem  with 
the  accuracy  of  the  documents,  but  take  issue  with  the  way  procedures  are  written. 

•  “Make  work  steps  easier  to  understand  and  make  steps  in  a  more  logical  order  to 
accomplish  work  more  efficiently” 

•  Involve  mechanics  with  the  initial  engineer/designer  so  the  wording  is  more 
understandable” 

•  “Have  experienced  mechanics  work  with  tech  writers  to  be  more  articulate  when 
describing  work-steps” 

•  “To  really  appreciate  the  problem,  pick  something  and  go  to  a  manual  and  see  the  length 
of  time  it  takes  to  figure  out  what  to  do” 

•  Tech  pubs  should  refer  to  maintenance  departments  and  establish  a  group  to  improve 
these  items  together” 

•  “Put  the  guy  in  some  coveralls  and  have  them  work  on  the  A/C  before  sitting  down  and 
writing  the  tech  pubs” 
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The  following  groups  of  comments  refer  to  the  general  level  of  detail  used  in  diagrams  and 
illustrations. 

•  “Quit  putting  a  small  picture  on  a  page  and  a  lot  of  numbers  referring  to  parts,  you  need  a 
magnifying  glass  to  see  where  the  arrows  are  directed” 

•  “Show  more  detail  on  a  components  and  systems” 

•  “Outline  the  steps  more  clearly,  and  if  an  item  appears  in  the  IPC,  it  should  be  mandatory 
to  reference  it  in  the  manual” 

Finally,  the  remaining  comments  refer  to  the  speed  at  which  manuals  are  updated  or  the  speed  of 
the  revision  process  after  problems  are  identified. 

•  “Quicker  corrections  to  the  manuals  when  I  point  out  the  errors” 

•  “Update  illustrations  and  diagrams,  replace  old  tech  data” 

•  “Listen  when  someone  reports  an  error” 

•  “Even  when  a  problem  is  reported  it  doesn’t  seem  to  ever  be  corrected” 

3.4.3  Important  Note. 

If  the  results  of  this  survey  were  limited  to  an  assessment  of  the  quality  of  technical  manuals,  the 
current  system  of  documentation  would  appear,  from  the  perspective  of  the  users,  to  be 
successful.  With  only  a  few  exceptions,  technicians  consider  manuals  informative,  consistent, 
and  accurate.  The  manufacturers  put  considerable  effort  into  verifying  the  accuracy  of  facts  and 
formatting  of  technical  documents,  and  the  technician’s  rate  them  high  in  those  areas.  If  the 
assessment  stopped  there,  the  results  would  indicate  no  need  for  improvement.  However,  when 
asked  about  the  ease  of  use,  efficiency,  and  suitability  of  the  procedural  information  contained  in 
the  maintenance  manual,  a  different  pattern  begins  to  emerge.  The  user’s  complaints,  as 
reviewed  above,  identify  usability  as  a  significant  issue.  This  likely  reflects  the  extent  to  which 
manufacturers  fail  to  take  into  account  the  way  technicians  use  maintenance  documents  and 
perform  aircraft  maintenance. 

3.5  INTERVIEW  RESULTS. 

Many  of  the  issues  voiced  during  the  site  visits  and  interviews  centered  on  the  difficulties 
technicians  have  locating  needed  information  in  the  maintenance  manual.  When  asked  for  a 
subjective  estimate  of  the  total  job  time  spent  searching  for  information,  most  technicians 
reported  that,  depending  upon  the  job,  as  much  as  50%  of  their  time  might  be  spent  searching  for 
information.  The  most  common  problems  reported  were  determining  effectivity  and  locating 
relevant  procedures  within  the  ATA  chapter  framework. 

3.5.1  Effectivity. 

The  aircraft  manufacturer  and  associated  parts  suppliers  are  constantly  working  to  improve  their 
respective  products.  Engineering  or  design  improvements  are  incorporated  into  the  production  of 
the  aircraft  as  they  become  available.  Consequently,  two  aircraft  of  the  same  model  and  type  can 
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differ  with  regard  to  specific  parts  and  components.  This  propensity  for  change  continues  once 
the  aircraft  is  put  into  service,  i.e.,  vendors  or  suppliers  may  go  out  of  business,  engineering 
changes  and  service  bulletins  may  lead  to  safety  or  regulatory-related  changes,  and  modifications 
may  be  made  to  upgrade  avionics  or  optional  equipment.  The  maintenance  documentation  must 
contain  information  specifying  the  parts  and  components  unique  to  each  aircraft. 

As  mentioned  in  Phase  1,  manual  writers  must  provide  references  for  all  of  the  engineering 
changes,  service  bulletins,  optional  equipment,  and  superceded  parts.  When  a  technician  begins 
a  maintenance  task,  they  must  first  establish  the  effectivity  of  all  options  that  apply  to  the  aircraft 
they  are  working  on,  as  well  as  the  status  of  any  modifications  or  service  bulletins.  Usually, 
technicians  must  search  through  all  of  the  possible  options  referencing  the  aircraft  tail  number 
against  the  parts  list.  Technicians  repeatedly  mentioned  difficulty  in  determining  effectivity  of 
parts  and  systems.  As  noted  above,  technicians  can  spend  as  much  as  50%  of  the  total  task  time 
determining  effectivity. 

3.5.2  Navigating  the  ATA  Framework. 


The  original  ATA  specification  100,  and  the  new  zspec  2200  that  replaces  it,  were  developed  to 
provide  standards  for  the  format  and  layout  of  technical  manuals.  In  addition  to  page  formatting 
and  text  layout  guidelines,  the  ATA  specification  details  an  outline  for  the  organization  of 
material  within  the  manual.  As  part  of  the  ATA  standard,  manufacturers  agree  to  conform  to  a 
common  industry  standard  for  chapter  organization.  For  example,  landing  gear  information  will 
always  be  located  in  chapter  32,  and  flight  controls  will  always  be  detailed  in  chapter  27. 
Standardization  minimizes  the  time  technicians  need  to  learn  to  use  new  manuals  and  makes  it 
easier  for  them  to  switch  from  one  manual  to  another.  This  standardization  improves  manual 
usability  and  user  experience  with  manuals. 

However,  even  when  using  the  framework  of  the  ATA  specification,  there  are  occasions  when 
the  manual  writer  must  make  decisions  regarding  the  best  location  for  component  description  or 
procedure  details.  For  example,  maintenance  information  pertaining  to  the  wings  is  located  in 
chapter  57  and  fuel  systems  are  located  in  chapter  28.  Because  fuel  tanks  are  located  in  the 
wings,  interfacing  parts  belonging  to  the  wing  may  appear  in  diagrams  of  the  fuel  system  but  only 
be  referenced  in  the  manual  chapter  dealing  with  the  wing.  In  this  case,  the  technician  must 
determine  which  system  each  component  belongs  to  and  where  the  relevant  information  can  be 
found.  These  organization  decisions  are  common  to  all  parts  of  the  aircraft  and  must  be  made 
throughout  the  manual.  Because  there  is  some  leeway  in  the  organization  of  information  within 
the  chapter  structure,  writers  may  make  decisions  based  on  engineering  classification  of  systems 
or  ease  of  writing.  While  the  design  and  manufacturing  processes  may  suggest  natural  answers 
to  these  questions,  the  response  of  the  technicians  indicates  that  the  organization  choices  made  by 
writers  often  creates  confusion  and  difficulty  for  technicians  when  attempting  to  find 
information. 

3.5.3  Design  of  Procedures. 

The  third  issue  raised  during  site  interviews  was  the  ordering  and  breakdown  of  procedural  steps. 
Rather  than  listing  the  same  procedural  steps  every  time  a  similar  task  must  be  accomplished. 
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most  manuals  will  list  them  once,  and  call  them  out  when  applicable.  This  subtask  call-out 
arrangement  reduces  overall  manual  size,  ensures  the  consistency  of  information,  and  increases 
efficiency  of  document  revisions.  If  a  procedure  must  be  revised,  it  is  easier  to  change  one 
instance  rather  than  multiple  instances  of  that  information.  In  a  paper-based  manual,  the  subtasks 
referenced  in  a  maintenance  procedure  may  be  distributed  throughout  the  manual,  forcing  the 
technician  to  jump  from  one  section  to  another.  In  some  cases,  a  procedure  will  call  for  only 
selected  steps  of  a  given  subtask,  further  fragmenting  the  flow  of  procedural  steps.  This  subtask 
distribution  is  well  suited  to  a  linked,  electronic  document  that  allows  the  user  to  follow  links  to 
each  subtask  and  then  return  to  the  original  procedure.  In  paper  or  microfilm  formats,  the 
technician  must  manually  search  for  and  retrieve  the  subtask  information.  The  time  spent 
searching  for  relevant  subtasks  and  the  disruption  of  workflow  increases  the  likelihood  that  a 
technician  may  miss  steps  or  lose  their  place  within  the  procedure. 

4.  DISCUSSION. 

Before  discussing  the  major  findings  of  this  study,  it  is  important  to  first  review  the  purpose  of 
the  investigation.  The  primary  object  of  Phase  2  was  to  assess  the  degree  of  error  present  in 
fielded  manuals  and  to  assess  their  usability  from  the  perspective  of  the  user.  The  following 
discussion  is  organized  into  four  main  sections  reviewing  the  validity  of  survey  responses, 
findings  pertaining  to  errors  in  the  manuals,  quality  of  manuals,  usability  ratings,  and  the 
potential  impact  of  each  of  these  factors  on  safety. 

4.1  RESPONSE  VALIDITY. 

In  order  to  determine  the  degree  to  which  the  survey  respondents  were  representative  of  the 
larger  population  of  aircraft  technicians,  a  series  of  questions  regarding  maintenance  safety 
behaviors  were  included.  These  questions  were  adapted  from  a  previous  survey  conducted  for 
the  Australian  Transportation  Safety  Bureau  (ATSB),  by  Hobbs  and  Williamson  [7],  of  safety 
behaviors  of  maintenance  technicians.  Survey  data  is  subject  to  a  variety  of  potential  biases. 
Participants  may  bias  their  responses  to  satisfy  what  they  believe  the  research  objective  to  be,  in 
fear  of  how  it  may  affect  their  employment,  or  how  it  might  reflect  on  them  or  their  profession. 
Although  it  is  not  possible  to  eliminate  all  sources  of  response  bias,  one  can  gauge  the  unique 
bias  of  a  group  by  comparing  responses  with  results  of  similar  studies.  Differences  in  the 
response  of  two  groups  to  similar  questions  may  suggest  the  presence  of  unique  response  biases. 
A  comparison  of  the  results  of  the  present  study,  with  those  of  Hobbs  and  Williamson,  indicates 
no  marked  differences  in  response  patterns  between  the  two  populations  of  technicians.  For 
example,  in  Hobbs  and  Williamson’s  survey,  34%  percent  of  technicians  reported  occasionally, 
often,  or  very  often  failing  to  refer  to  the  parts  catalogue  when  selecting  parts.  In  response  to  the 
same  question,  33%  of  technicians  who  participated  in  this  survey  made  the  same  responses. 
Likewise,  91%  percent  of  respondents  in  Hobbs  and  Williamson’s  survey  report  never  or  rarely 
failing  to  refer  to  the  maintenance  manual  on  an  unfamiliar  job.  Eight-six  percent  of  respondents 
in  this  survey  responded  in  a  similar  fashion.  The  similarity  of  responses  is  notable,  considering 
the  differences  in  nationality  and  regulatory  controls  to  which  each  is  subjected.  More 
importantly,  the  results  suggest  that  the  replies  and  experiences  of  our  pool  of  respondents  is 
representative  of  aircraft  maintenance  personnel  in  general. 
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4.2  SUMMARY  OF  MAIN  FINDINGS. 


Because  of  concerns  about  the  potential  impact  on  safety,  manual  error  has  received  greater 
attention  recently.  This  is  partly  in  response  to  a  number  of  aircraft  accidents  attributed,  at  least 
in  part,  to  maintenance  errors.  In  the  absence  of  hard  data,  the  magnitude  of  this  problem  could 
not  be  estimated.  This  project  represents  an  attempt  to  determine  the  extent  and  potential  impact 
of  errors  in  aviation  maintenance  manuals. 

In  general,  the  results  indicate  that  the  number  of  errors  in  fielded  manuals  may  be  relatively  low 
given  that  most  respondents  reported  rarely  or  never  finding  errors  in  text  (54%),  illustrations 
(63%)  or  diagrams  (66%).  One  might  be  alarmed  to  see  that  roughly  35%  to  45%  of  respondents 
reported  occasionally  or  often  finding  errors  in  these  areas.  However,  many  of  these  errors,  such 
as  misspellings  or  format  errors,  would  have  little  impact  on  maintenance  or  aircraft  safety. 

Comparisons  of  errors  found  in  text  versus  diagrams  shows  that  errors  appear  to  be  more 
frequent  in  text.  This  does  not  mean  that  textual  information  is  necessarily  less  accurate  than  the 
information  presented  in  diagrams  or  illustrations.  This  pattern  more  likely  reflects  the  fact  that 
text  procedures  represent  a  proportionally  larger  part  of  the  manual,  or  it  may  be  the  result  of  the 
difficulty  of  putting  complex  procedures  and  descriptions  into  words  versus  presenting  that 
information  in  graphical  form. 

Interestingly,  comparisons  of  large  and  smaller  FAR  Part  25  aircraft  indicate  that  the  only 
significant  differences  were  related  to  the  perceived  frequency  of  text  errors,  with  error  being 
reported  more  frequent  in  smaller  FAR  Part  25  aircraft.  It  might  be  expected  that  developing 
manuals  for  large  transport  aircraft  would  in  some  way  be  different  from  other  FAR  Part  25 
aircraft  because  of  the  level  of  customization  and  system  complexity.  While  there  are  differences 
in  the  user’s  perception  of  error  between  the  manufacturers  of  these  aircraft,  the  degree  of 

similarity  in  responses  is  surprising,  in  light  of  the  lack  of  common  error  controls  being 
employed. 


User  feedback  is  one  of  the  primary  factors  influencing  the  development  of  technical 
documentation.  Each  company  employs  its  own  set  of  procedures  to  develop  a  manual  they  think 
users  will  accept.  In  addition  to  user  satisfaction,  these  procedures  must  strike  a  balance  between 
a  number  of  competing  demands  including  manual  development  costs,  time  constraints, 
regulatory  requirements,  and  safety.  Each  manufacturer  has  attempted  to  achieve  a  balance 
among  these  competing  demands,  and  the  variety  of  approaches  adopted  reflects  the  uniqueness 
of  those  solutions.  Considering  that  the  majority  of  users  (56%)  rate  the  quality  of  manuals  as 
being  good  to  veiy  good,  it  would  appear  that  the  demands  of  users  are  being  met. 

4.3  QUALITY  OF  MANUALS 

If  the  manufacturers  perception  of  the  quality  of  their  manuals  is  based  on  the  feedback  obtained 
from  users,  this  perception  may  be  in  error  because  they  do  not  know  the  number  of  errors  that  go 
unreported.  More  importantly,  it  places  a  greater  emphasis  on  factual  and  typographic  error  (e.g., 
incorrect  part  numbers  and  misspellings)  while  providing  little  information  about  the  usability  of 
the  document  and  procedures.  It  is  quite  possible  for  a  document  to  be  factually  correct  and  free 
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of  spelling,  grammar,  and  formatting  errors,  yet  fall  short  of  the  goal  of  good  usability.  For 
example,  Patankar  and  Kanki  [8]  outline  changes  in  the  engine  change  procedure  of  a  Boeing  737 
that  resulted  in  a  significant  decrease  in  total  job  hours.  The  factual  information  remained  the 
same,  but  improvements  in  the  usability  of  the  document  resulted  in  a  more  efficient,  and 
presumably  safer  procedure. 

The  number  of  disagree  and  strongly  disagree  responses  to  the  following  questions  reflect  a 
problem  with  manual  usability:  “the  manual  describes  the  best  way  to  do  a  procedure”  and  “the 
manual  writer  understands  the  way  I  do  maintenance.”  What  is  of  issue  here  is  whether  the 
procedural  steps  can  be  easily  followed,  whether  procedures  progress  in  a  logical  and  efficient 
manner,  and  whether  the  task  instructions  take  into  account  possible  extenuating  circumstances 
or  the  difficulty  of  certain  tasks.  When  writing  a  task  procedure,  the  writer  must  decide  what 
order  to  follow,  the  number  and  extent  of  warnings  to  include,  how  much  background 
information  to  include,  and  how  much  detail  to  provide.  In  many  cases,  these  choices  are  made 
based  on  either  design  standards  or  personal  experience.  Ideally,  these  decisions  should  be 
guided  by  observation  of  individuals  performing  the  task  and/or  feedback  from  technicians.  The 
farther  removed  the  writer  is  from  the  task  demands,  the  less  aware  they  will  be  of  potential 
usability  problems. 

Clearly,  there  is  a  need  to  provide  maintenance  technicians  with  documentation  that  not  only 
conveys  the  technical  information  necessary  to  maintain  an  aircraft,  but  also  presents  it  in  such  a 
way  that  best  matches  the  way  technicians  do  their  job.  All  manufacturers  have  systems  in  place 
to  gather  feedback  from  operators  about  the  accuracy  of  manuals,  but  gathering  data  about  user 
perceptions  of  manual  usability  has  not  traditionally  received  as  much  attention.  This  type  of 
feedback  requires  more  detailed  communication  from  the  operator  and  a  greater  commitment 
from  manufacturers  to  gather  that  feedback.  The  majority  of  current  response  forms  and 
suggestion-reporting  methods  are  tailored  more  toward  the  traditional  idea  that  errors  are  being 
limited  to  incorrect  information.  When  evaluation  of  manual  quality  is  extended  beyond 
information  error  into  issues  of  document  usability,  a  reactive  approach  becomes  less  effective 
because  user’s  are  unlikely  to  report  usability  problems  and  even  less  likely  to  make 
nontechnicians  able  to  understand  those  problems.  Users  may  not  take  time  to  provide  feedback, 
may  fail  to  articulate  the  problem  adequately,  or  the  writer  may  fail  to  appreciate  the  cause  or 
extent  of  the  user’s  difficulty.  Analysis  of  survey  responses  to  the  question  “How  often  do  you 
report  errors  when  you  find  them?”  reveals  that  in  many  cases  technicians  simply  do  not  take  the 
time  to  report  errors.  Because  users  represent  the  last  line  of  defense  in  ensuring  manual  quality 
and  accuracy,  any  failure  to  report  an  identified  error  can  be  considered  a  degradation  of  the 
entire  system.  The  goal  then  becomes  one  of  determining  why  users  might  fail  to  report  errors 
and  how  the  reporting  system  might  be  improved.  In  both  the  on-site  interviews  and  survey 
comments,  technicians  frequently  commented  that  they  received  little  feedback  or  saw  no 
changes  to  the  manual  as  a  result  of  their  suggestions.  If  there  is  a  perception  that  no  one  is 
listening  to  their  input,  the  technicians  will  simply  stop  providing  feedback.  What  remains 
unclear  is  where  this  breakdown  is  occurring.  Supervisors  may  fail  to  pass  information  on  to  the 
writers  and  engineers  within  their  operator,  those  writers  and  engineers  may  fail  to  pass  it  on  to 
the  manufacturer’s  service  representative,  or  the  manufacturer  may  fail  to  adequately  respond  to 
the  input  of  the  service  representative.  In  many  cases,  a  change  request  may  be  denied  but  the 
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reporting  technician  is  left  unaware  of  the  reasons  for  this  decision.  If  technicians  believe  that 
their  comments  were  ignored,  they  will  be  less  likely  to  report  problems  in  the  future.  In  order 
for  the  reporting  process  to  work,  a  greater  commitment  is  required  from  both  the  operator  and 
the  manufacturer  to  ensure  that  identified  problems  are  reported  in  sufficient  detail,  and  a  follow¬ 
up  response  is  provided  to  the  reporting  technician  even  when  no  changes  are  made. 

Potential  consequences  of  usability  problems  in  technical  documents  include  the  safety,  speed, 
and  cost  of  aircraft  maintenance.  In  a  complex,  time-limited  environment  like  aircraft 
maintenance,  these  factors  are  interdependent.  Every  time  a  technician  has  difficulty  finding 
information  in  the  manual,  the  speed  of  maintenance  is  slowed  and  revenue  is  lost.  Because  of 
the  financial  impact  and  schedule  demands,  the  pressure  to  complete  work  quickly  can  push 
individuals  to  take  shortcuts.  If  the  technician  has  difficulty  understanding  the  wording  of  a 
manual  procedure,  they  may  be  motivated  to  devise  an  alternate  method  of  completing  that  task 
or  ask  another  technician  for  help.  With  the  average  level  of  experience  represented  in  this 
survey  (13  years),  this  may  not  result  in  a  problem.  However,  whenever  users  are  pushed  to  work 
around  usability  problems,  the  potential  for  safety  violations  increases. 

Some  aspects  of  manual  usability,  such  as  formatting,  consistency,  and  reading  level,  can  be 
controlled  through  standardization  and  guidelines.  The  need  for  clarity  and  consistency  across 
manuals  has  been  the  motivation  for  the  ATA  guidelines  regarding  the  layout  and  organization  of 
maintenance  documentation.  The  ATA  Spec  100//spec  2200  has  been  an  effective  means  of 
establishing  a  baseline  for  manual  quality  and  structure.  The  pattern  of  responses  to  survey 
questions  pertaining  to  manual  quality,  consistency,  and  ease  of  use  suggest  that  the  standards 
adopted  by  the  ATA  have  been  effective.  The  use  of  consistent  style  and  formatting  is  one 
important  aspect  of  document  usability.  However,  within  the  framework  of  the  ATA 
specification,  the  writer  must  make  decisions  about  details  such  as  the  ordering  of  procedural 
steps,  the  wording  of  procedures,  the  use  of  illustrations,  and  level  of  detail.  Questions  about 
wording  and  sentence  structure  can  be  addressed  with  the  use  of  restricted  vocabularies  or  style 
guides,  but  the  unique  nature  of  each  maintenance  procedure  limits  the  usefulness  of  any 
checklist  or  standard  method  for  describing  a  procedure.  Writers  and  engineers  must  rely  on 
their  writing  experience  and  knowledge  of  the  aircraft  systems  when  deciding  how  best  to 
describe  a  procedure.  Likewise,  a  technician  will  rely  on  their  maintenance  experience  and 
knowledge  of  aircraft  systems  when  interpreting  and  applying  maintenance  information. 
Unfortunately,  the  experience  and  knowledge  of  the  writer  and  engineer  may  be  quite  different 
from  the  experience  and  knowledge  of  the  technician.  This  potential  mismatch  can  result  in 
other  usability  problems  in  technical  documentation.  At  least  one  participating  manufacturer  has 
started  to  address  some  of  these  issues  by  validating  maintenance  manual  procedures. 
Unfortunately,  the  responses  included  in  the  current  survey  do  not  include  enough  references  to 
validated  manuals  to  ascertain  the  effects  on  user  perception. 


In  order  to  understand  how  usability  problems  occur,  it  is  necessary  to  understand  the  cognitive 
processes  involved  in  reading  and  applying  information.  Hayes  and  Flower  [9]  provide  the 
following  outline  (see  figure  10)  adapted  from  Thibadeau  and  Just  [10]  of  the  cognitive  process 
of  reading  to  comprehend  information.  The  process  of  reading  can  be  thought  of  as  a  flow  from 
the  basic  level  of  letter  and  word  identification  to  the  most  advanced  level  of  understanding  the 
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intent  of  the  presented  message  and  applying  the  information  it  contains.  Progression  along  this 
continuum  from  decoding  of  words  to  inference  of  intent  is  a  move  from  text-centered,  to 
information-  and  reader-centered  processes  [11].  Any  written  passage  can  be  evaluated  for  how 
well  it  performs  at  any  level  along  this  continuum,  but  the  criteria  for  evaluation  changes  in  the 
progression  from  text-  to  reader-centered  stages. 


FIGURE  1 0.  THE  PROCESS  OF  READING  TO  COMPREHEND 
_  (Adapted  from  references  9  and  10) 

In  the  early  stages  of  the  cognitive  model,  the  reader  identifies  words,  applies  rules  of  grammar, 
and  combines  sentences  to  create  a  semantic  interpretation  of  the  text.  This  is  a  rule-driven 
process,  based  on  the  reader’s  knowledge  of  the  language.  Errors  identified  during  these  stages 
include  misspelling,  incorrect  use  of  punctuation,  and  inappropriate  usage  of  words.  These 
mechanistic  problems  can  be  eliminated  through  spelling  and  grammar  checking  and  using  style 
guides  or  restricted  vocabularies.  The  impact  of  errors  at  these  stages  on  the  readers 
comprehension  will  vary  with  the  severity  of  the  error  and  the  expertise  of  the  reader.  Certainly, 
a  typographic  error  might  result  in  a  safety  problem  if  that  error  were  to  occur  in  a  part  number,  a 
tolerance  level,  a  pressure  level,  or  torque  value.  However,  it  is  more  likely  that  typographic  and 
grammatical  errors  will  be  seen  as  annoyances  rather  than  safety  problems. 
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Stages  1  through  3  of  the  model  focus  on  the  mechanical  structure  of  the  text,  while  stages  4,  5, 
and  6  shift  the  focus  beyond  the  text  itself  to  the  topic  area  the  text  is  referencing.  The  majority 
of  procedures  currently  in  place  are  meant  to  limit  problems  related  to  the  early  stages  of  reading 
cognition.  In  order  to  address  problems  related  to  the  topic  area,  requires  an  appreciation  of  the 
cultural  and  institutional  environment  associated  with  aircraft  maintenance.  It  is  at  this  stage  that 
many  of  the  problems  related  to  language  translation  occur.  Even  differences  within  the  same 
language  can  cause  problems  when  addressing  topic  area  conventions.  For  example,  the  British 
and  American  use  of  the  terms  “torch”  and  “flashlight.”  A  British  writer’s  description  of  a  fuel 
tank  inspection  procedure  could  potentially  lead  to  disaster  for  an  American  technician  if  they 
fail  to  understand  the  usage  of  the  word  torch. 

Finally,  stages  7  and  8  involve  the  combination  of  semantic  knowledge,  conventions  of  the  topic 
area,  and  the  reader’s  expectations  to  arrive  at  an  interpretation  of  the  writer’s  intent.  If  the 
writer  fails  to  account  for  the  presuppositions  of  the  reader,  all  other  aspects  of  the  text  can  be 
accurate,  yet  still  be  misdirected.  At  best,  the  reader  may  simply  discount  the  importance  of  the 
writer’s  point  of  view.  At  worst,  the  reader  may  misunderstand  the  point  of  view  of  the  writer 
and  perform  a  procedure  in  a  manner  contrary  to  what  the  writer  intended.  Minimizing  the 
possibility  of  failure  at  any  point  along  the  continuum  requires  a  combination  of  error-checking 
and  proofreading  procedures,  along  with  clear  communication  between  writers  and  readers 
throughout  the  writing  process  and  beyond. 

4.4  FUTURE  USABILITY  ISSUES. 

The  responses  included  in  this  survey  apply  primarily  to  paper-based  manual  formats.  There  is  a 
slow  but  steady  trend  to  replace  these  manuals  with  computer-based  maintenance  information. 
The  move  from  paper-based  to  electronic  manuals  will  address  many  of  the  usability  problems 
reported  by  technicians.  Electronic  manuals  can  be  updated  and  distributed  quickly,  take  up  less 
space,  and  can  make  searching  for  information  much  easier.  In  addition,  they  may  include  text, 
audio,  and  video  presentations  to  provide  a  more  detailed  description  of  procedures.  Even  with 
the  potential  benefits  of  moving  to  electronic  manuals,  the  change  of  format  introduces  new 
usability  issues.  These  include  the  design  of  the  interface,  search  functions,  size  and  readability 
of  video  displays,  document  navigation,  and  equipment  reliability.  Every  subsequent 
development  in  the  delivery  of  technical  information  will  also  require  careful  scrutiny  of  usability 
concerns. 

5.  IMPLICATIONS. 

The  results  of  the  survey  suggest  that  the  users  of  aviation  technical  manuals  generally  perceive 
the  manuals  favorably.  They  did  not  report  significant  problems  with  the  accuracy  or  general 
quality  of  the  documents.  The  main  criticism  shared  by  most  users  pertains  to  usability.  More 
specifically,  users  comment  that,  in  many  cases,  the  document  procedures  are  inefficient  and/or 
fail  to  consider  the  demands  of  the  maintenance  environment.  Considering  that  these  concerns 
were  shared  by  technicians  using  manuals  from  a  diverse  set  of  manufacturers,  it  is  expected  that 
improving  the  usability  of  technical  documents  may  have  a  broad  range  of  benefits  ranging  from 
improved  user  satisfaction  to  improved  safety  and  savings  in  maintenance  costs.  The  goal  of 

Phase  3  of  this  effort  will  be  to  outline  recommendations  and  procedures  to  improve  document 
usability. 
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APPENDIX  A— SAMPLE  SURVEY  QUESTIONNAIRE 


FAA  Mechanic  Questionnaire 


Please  list  the  aircraft  you  currently  work  on  and  indicate  the  length  of  time  you  have  worked  on  them: 


Aircraft 

Name 

Specialty  Area  (e.g.  wiring,  avionics,  engines, 

ALL) 

Less  than  3 
months 

3-6 

months 

6-12 

months 

1  -3 

years 

More  than  3 
years 

Please  list  the  TWO  aircraft  you  have  worked  on  the  MOST  (Aircraft  A  and  B).  Then,  answer  the  following  questions  based  on  those  aircraft. 
Aircraft  A: _  Aircraft  B: _ 


Aircraft  A  Aircraft  B 


In  a  typical  week,  how  often  do  you  use  the  following  manuals? 

<  5  hrs 

6-10  hrs 

10-30  hrs 

>  30  hrs 

<  5  hrs 

6-10  hrs 

10-30  hrs 

>30  hrs 

Aircraft  Maintenance  Manual  (AMM) 

1 

2 

3 

4 

1 

l 

J 

4 

Component  Maintenance  Manual  (CMM) 

2 

3 

4 

t 

2 

3 

4 

Engine  Cleaning,  Inspection,  and  Repair  Manual  (CIR) 

1 

2 

3 

4 

l 

2 

3 

4 

Engine  Shop  Manual  (EM) 

1 

> 

i 

4 

J 

2 

3 

4 

Fault  Isolation  Manual  (FIM) 

1 

2 

3 

4 

I 

> 

3 

4 

Service  Bulletin  (SB) 

J 

2 

3 

4 

1 

2 

3 

4 

Structural  Repair  Manual  (SRM) 

1 

2 

5 

4 

I 

2 

3 

4 

Aircraft  Illustrated  Parts  Catalog  (I PC) 

1 

■> 

$ 

4 

I 

2 

3 

4 

Wiring  Manual  (WM) 

1 

2 

3 

4 

\ 

2 

3 

4 

Task  Cards 

1 

2 

3 

4 

I 

2 

3 

4 

Company  (or  in-house)  developed  maintenance  aids 

1 

2 

3 

4 

f 

2 

3 

4 

Other 

J 

2 

3 

4 

t 

: 

3 

4 

How  often  have  you  found  an  error  Never 

Very 

Aircraft  A 
Occasionally 

Often 

Very 

Never  Very 

Aircraft  B 
Occasionally 

Often 

Very 

in: 

Text  Descriptions  or  Procedures  I 

Rarely 

2 

. 

4 

Often 

5 

Rarely 

1  2 

3 

"4 . 

Often 

'  5  ~~ 

Illustrations  1 

2 

3 

4 

5 

1  2 

3 

4 

$ 

Diagrams  1 

2 

3 

4 

5 

^  2 

3 

4 

? 

How  would  you  rate  the  following  aspects  of  the  Aircraft 
Maintenance  Manuals? 

Very 

Poor 

Poor 

Aircraft  A 
Neutral 

Good 

Very 

Good 

Very 

Poor 

Poor 

Aircraft  B 

Neutral 

Good 

Very 

Good 

Usefulness  of  information 

1 

2 

3 

4 

5 

1 

2 

3 

4 

5 

Quality  of  the  illustrations 

1 

2 

3 

4 

5 

1 

2 

3 

4 

5 

Quality  of  diagrams 

1 

2 

3 

4 

5 

1 

2 

3 

4 

5 

Clarity  of  text  descriptions 

1 

y 

3 

4 

5 

1 

7 

3 

4 

5 

Aircraft  A 

Aircraft  B 

General  Satisfaction 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

Strongly 
Disagree 
.  1 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

l  find  the  manual  information  to  be 
very  informative. 

1 

2 

3 

4 

5 

2 

3 

4 

5  ••• 

There  is  never  enough  information  in 
the  manual  when  I  need  it. 

I 

2 

3 

4 

5 

1 

■» 

3 

4 

5 

Using  these  manuals  is  frustrating. 

1 

2 

3 

4 

5 

1 

2 

3 

4 

5 

Sometimes  I  fed  like  I  need  a  manual 
to  understand  the  manual.  - 

t 

2 

3 

4 

5 

1 

2 

3 

4 

5 

The  information  in  the  manual  is 
inconsistent 

1 

2 

3 

4 

5 

I 

2 

3 

4 

5 

The  maintenance  manual  describes  the 
easiest  way  to  conduct  each  procedure 

1 

o 

3 

4 

5 

1 

2 

3 

4 

5 

The  writers  of  the  manual  understand 
the  way  I  do  maintenance  procedures. 

I 

2 

3 

4 

5 

1 

y 

3 

4 

5 

The  manuals  help  me  to  be  more 
productive. 

1 

2 

3 

4 

5 

1 

2 

3 

4 

5 

The  manuals  are  easy  to  use. 

l 

2 

3 

4 

5 

1 

2 

3 

4 

5 

The  manuals  were  easy  to  learn  how  to 
use. 

1 

•> 

3 

4 

5 

1 

2 

3 

4 

5 

I  am  satisfied  with  the  manuals. 

I 

2 

3 

4 

5 

1 

2 

3 

4 

5 

I  easily  remember  how  to  use  the 
manuals. 

1 

1 

3 

4 

5 

1 

1 

3 

4 

5 

An  inexperienced  mechanic  could 
repair  the  aircraft  using  just  the  manual 

l 

2 

3 

4 

5 

1 

2 

3 

4 

5 

A-l 


Did  you  receive  training  on  how  to  use  the  manuals? 


Aircraft  A  Aircraft  B 

YES  :.><0  YES-  NO 


Did  you  find  the  training  to  be  adequate?  YES  NO 

Do  you  receive  training  on  the  manuals  as  updates  are  made?  YES  NO 


YES  ISO 


At  work  in  the  last  year  or  so,  how  often  have  you: 

Be^ri  misled  by  confusing  documentation 

Not  referred  to  the  maintenance  manual  or  other  approved  documentation  on  a  familiar  job 
Not  referred  to  the  maintenance  manual  or  other  approved  documentation  on  a  Unfainiliar  job 
Not  referred  to  the  parts  catalogue  when  selecting  a  part  * 

Done  a  job  a  better  way  than  that  in' the  manual 

Adjusted  or  rigged  a  system  incorrectly  because  the  documentation  was  unclear  or  misleading 
Assembled  a  component  or  system  incorrectly  because  the  documentation  was  unclear  of 
misleading 


Never  Very  Occasionally 

Rarely  __ 

!  3 . : "J . 

l  2  3 

2  E  W^' 

i  .  2  3 

i  ■'  ■'  SI?  !•..  •  <-3  " 

1  2  3 

'••I  '  2'  "  3 


Often 

IIP; 

4 

mm 

4 

4 


Very 

often 

.  '5"  "' 

. 5 . 

V'S.:' 

. 5 

5 

5 

. 


If  you  have  discovered  a  poorly  worded  or 
unclear  procedure,  how  often  did  it  result  in 

•A  part  being  damaged 

The  need  for  help  from  another  mechanic 

A  mechanic  finding  their  own  way  to  do^N^  N 
procedure 

An  aircraft  being  released  in  an  unairworthy 
condition 

A  mechanic  being  blamed  for  a  mishap 


Never  Very  Rarely  Occasionally  Often  Very  Often 


Never  Very  Rarely  Occasionally  Often  Very  Often 


In  genera],  when  a  problem  is  discovered  in  a  l  r . 3 . mTW . ■  ■  .3^ 

manual,  how  often  is  it  reported? 


What  manual  format  do  you  use  most  often? 

_ Paper 

_ Microfilm 

_ _  Microfiche 

_ CD-ROM 

_ Portable  Maintenance  Aid 

_ Other  (specify) _ 


Which  manual  format  do  you  prefer?  _ ; _  Why? 


What  suggestions  do  you  have  for  improving  maintenance  manuals? 


-BIOGRAPHICAL  INFORMATION- 


Company/Location:  _ 

Education  level: 

_ High  school  graduate 

_ Some  college 

_ Bachelor's  degree 

_ Graduate  degree 

_ Other  (specify) _ . 


Maintenance  Training  (check  all  that  apply): 

_ Technical  School  _ years 

_ _  Military _ years 

_ Other  _ years 

Type  of  Maintenance  Typically  Performed: 

_ Line  Maintenance 

_ _ Base  Maintenance 


Job  title:  _ 

FAA  Certification: 

_ None 

_ Yes  (specify)  _ 

Typical  work  hours: 

Start  time:  _  AM  PM 

End  time:  _  AM  PM 

Total  hours  per  week:  _ _ 

Gender:  M  F 

Year  of  birth: _ 

Years  in  current  position: _ 

Years  with  die  company: _ 
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APPENDIX  B— STATISTICAL  ANALYSIS  OF  LARGE  VERSUS  SMALLER  AIRCRAFT 


■1 

t-test  for  Equality  of  Means 

;Si^g>6e(2-feiIeag^ 

Me^Difretwco^^ 

Found  Error  in  Text 

-2.461 

295 

0.014 

-0.257 

Found  Error  in  Illustrations 

-0.891 

293 

0.374 

-0.083 

Found  Error  in  Diagrams 

-1.399 

292 

0.163 

-0.130 

Usefulness  of  Manuals 

1.182 

285 

0.238 

0.107 

Quality  of  Manuals 

-0.347 

295 

0.729 

-0.036 

Quality  of  Diagrams 

-0.435 

294 

0.664 

-0.044 

Clarity  of  Text  Descriptions 

2.317 

296 

0.021 

0.254 

Manuals  are  Informative 

2.012 

295 

0.045 

0.185 

There  is  Never  Enough  Information 

-3.363 

295 

— — - - - - - 

0.001 

-0.351 

Using  the  Manual  is  Frustrating 

-0.586 

295 

0.559 

-0.068 

1  Need  a  Manual  to  Understand  the  Manual 

-0.422 

293 

0673 

-0.049 

The  Manual  Information  is  Inconsistent 

-2.749 

296 

0006 

-0.273 

The  Manual  Describes  the  Easiest  Way  to  Do  a  Procedure 

0.273 

294 

0785 

0028 

The  Manual  Writer  Understands  How  1  do  Maintenance 

0.071 

296 

0.943 

0.007 

Manuals  Help  Me  To  Be  More  Productive 

0.669 

296 

0504 

0.070 

Manuals  are  Easy  to  Use 

0.623 

290 

0534 

0.062 

Manuals  are  Easy  to  Learn 

0.710 

295 

0479 

0.073 

1  am  Satisfied  with  the  Manual 

1.110 

291 

0268 

0.125 

lt‘s  Easy  to  Remember  how  to  use  the  Manual 

-1.302 

291 

0.194 

-0.122 

Received  Training  on  Manuals 

1.039 

299 

0.299 

0.057 

Training  was  Adequate 

1.938 

275 

0.054 

0.113 

Received  Additional  Training  with  Manual  Updates 

0.888 

295 

0375 

0.043 

Been  Misled  by  Manual  Information 

-0.576 

145 

0.565 

-0.074 

Not  used  a  Manual  on  a  Familiar  Job 

-0.883 

146 

0379 

-0.145 

Not  used  a  Manual  on  an  Unfamiliar  Job 

1.104 

146 

0.271 

0.137 

Not  Referred  to  IPC  When  Collecting  Parts 

4.439 

146 

0000 

0.653 

Done  a  Job  a  Better  Way  Than  in  the  Manual 

-2.327 

144 

0.021 

-0.358 

Adjusted  or  Rigged  Incorrectly  due  to  Unclear  Documentation 

-1.154 

144 

0250 

-0.168 

Assembled  Parts  Incorrectly  due  to  Unclear  Documentation 

-0.186 

145 

0.853 

-0.023 

Education  Level 

-1.952 

145 

0053 

-0.447 
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